Analysis of Physicians’ Probability Estimates of a Medical Outcome Based on a Sequence of Events

Key Points Question Are physicians able to accurately estimate the overall probability of a medical outcome resulting from 2 independent events? Findings In this survey study of 215 physicians, most respondents (78.1%) estimated the probability of a medical outcome resulting from a 2-step sequence to be greater than the probability of at least 1 of the 2 component events, a result that was mathematically incoherent (ie, formally illogical and mathematically incorrect). Meaning This study’s findings suggest that because many diagnostic and prognostic decisions require more than 1 step or the consideration of more than 1 probability, misestimation of the overall probability of success when 2 or more independent events are involved (termed the conjunction fallacy) is likely to be a source of diagnostic and prognostic error.

In our 3 studies 285, 496, and 341 invitations were issued to potential respondents respectively. The 3 survey websites were accessed by 73, 92, and 78 respondents respectively.
As mentioned in our method section, we did not collect data from a physician in the first study because that person did not attend deliveries. Thus 72 eligible people and 1 ineligible person accessed the website of the first study. The final number of eligible respondents in the 3 studies was 67, 84, and 64 respectively, for a response rate of 89%.
The reason the response rate was extremely high is that all participating physicians were in a panel administered by Reckner Healthcare for the purpose of responding to surveys. Thus, all participants had already volunteered to do such surveys for renumeration. After Reckner sent out their standard invitation to the appropriate group (obstetricians or pulmonologists), all eligible physicians in that group could have responded. However, we stopped enrollment before many of the panel members could respond. How many of the remaining non-enrollees wanted to respond is unknown. Thus, we cannot use as the denominator to calculate response rate the number of respondents who could have taken the survey, because we stopped access to the survey once our target census was reached.
We have no reason to believe that our responders are unrepresentative, but whether the incidence of the conjunction fallacy we found is representative of a broad selection of physicians will require confirmation in future studies. However, the large proportion of physicians committing the fallacy in our study suggests that even if the true proportion is smaller, it will still be a prevalent error.
used the "CHERRIES" guidelines checklist, which we present on the next page. At the top of the checklist are the following: the bibliographic information of the checklist, a link to the original publication, and the copyright and license information.
We begin with an example of the notice sent to prospective respondents. (The notices for the 3 studies varied in miniscule ways.) This is our response to item #7 in the checklist. Then we present the checklist. Then we present the consent form that contains some of the information requested in the checklist.
Notice sent to potential respondents: The healthcare industry welcomes your opinions, and we therefore invite you to participate in this upcoming research study: Compensation: $30 to be redeemed via a Visa® or Amazon® Reward card. As with all marketing research, there are a few questions we will ask before you start the survey to ensure that you meet the criteria for the study.
Reckner Healthcare is a leader in medical market research since 1991. We work with healthcare professionals to gather their expert opinions on a range of healthcare developments via online, phone, and in-person studies. Last year alone, we worked with 19,000 healthcare professionals and paid nearly $8 million in honoraria. More info is available at http://healthcaresurveys.reckner.com.
Thank you.

Your participation is voluntary.
Please consider the information carefully. Feel free to ask questions before making your decision whether or not to participate.

Purpose:
Learn how medical experts make decisions.

Procedures/Tasks:
You will make a series of probability estimates about an obstetric situation.
Duration: Today's task will take approximately 10 minutes.
You may leave the study at any time. If you decide to stop participating in the study, there will be no penalty to you. Your decision will not affect your future relationship with The Ohio State University.

Risks and Benefits:
The risks of this study are not greater than what you encounter in everyday life. There are no direct benefits to you as an individual, but your responses will help us learn more about how experts make decisions.

Confidentiality:
We will work to make sure that no one sees your online responses without approval. But, because we are using the Internet, there is a chance that someone could access your online responses without anyone's permission. In some cases, this information could be used to identify you. By law, payments to participants are considered taxable income.

Participant Rights:
You may refuse to participate in this study without penalty. If you choose to participate in the study, you may discontinue participation at any time without penalty. By agreeing to participate, you do not give up any personal legal rights you may have as a participant in this study.
This study has been determined to be exempt from IRB review.

Contacts and Questions:
For questions, concerns, or complaints about the study, or you feel you have been harmed as a result of study participation, you may contact Dr. Hal R. Arkes (arkes.1@osu.edu).
For questions about your rights as a participant in this study or to discuss other study-related concerns or complaints with someone who is not part of the research team, you may contact the Office of Responsible Research Practices at 1-800-678-6251 or hsconcerns@osu.edu.

Providing consent
I have read this page and I am aware that I am being asked to participate in a research study. I have had the opportunity to ask questions by contacting Dr. Arkes and have had them answered to my satisfaction. I voluntarily agree to participate in this study. I am not giving up any legal rights by agreeing to participate.
To print or save a copy of this page, select the print button on your web browser.
Please click the button below to proceed and participate in this study. If you do not wish to participate, please close out your browser window.

eAppendix 1. Analysis Comparing the Discrepancy Between the Conjunction Estimate and the Product of the 2 Components Among Physicians Who Committed Any Conjunction Fallacy vs Physicians Who Did Not
In Study 1 (brow presentation) we compared the discrepancy between the conjunction estimate and the product of the 2 components for those who committed some form of the conjunction fallacy and those who did not. The 50 respondents who succumbed to the fallacy had a mean discrepancy of 16.1%. The 17 who did not succumb had a mean discrepancy of 3.3%.
This difference was significant [t(58.4) = 5.23, P <.001], Cohen's d = 1.06, (95%CI = .47 -1.63). Not committing the conjunction fallacy was associated with a reduction of the discrepancy between one's estimate of the conjunction and the product of the 2 components.
In Study 2 (pulmonary nodule) we compared the discrepancy between the conjunction estimate and the product of the 2 components for those who committed some form of the conjunction fallacy and those who did not. The 73 respondents who succumbed to the fallacy had a mean discrepancy of 22.0%. The 11 who did not succumb had a mean discrepancy of 5.4%.
This difference was significant [t(53.1) = 7.74, P <.001], Cohen's d = 1.21, (95%CI = .54 -1.86). Not committing the conjunction fallacy was again associated with a reduction of the discrepancy between one's estimate of the conjunction and the product of the 2 components.
In Study 3 (debiasing brow presentation) We compared the discrepancy between the conjunction estimate and the product of the 2 components for those who committed some form of the conjunction fallacy and those who did not. The 45 respondents who succumbed to the fallacy had a mean discrepancy of 25.4%. The 19 who did not succumb had a mean discrepancy of only 0.4%. This difference was significant [t(62) = 6.44, P <.001], Cohen's d =1.76, (95%CI = 1.14 -2.37). Not committing the conjunction fallacy was again associated with a reduction of the discrepancy between one's estimate of the conjunction and the product of the 2 components.
We have added these analyses to show that not committing the conjunction fallacy results in a substantial improvement in the correspondence between one's conjunctive estimate and the product of the 2 components' estimates. The over-estimation of the conjunction relative to the product is reduced when the conjunction fallacy is absent. Our conjecture is the following. In Studies 1 and 3 there are two steps to consider, namely, conversion to OP and vaginal delivery. In Study 2 the first component represents a base rate, namely, the pretest probability that the patient has a malignancy. In some of their earliest judgment/decision making research, Kahneman and Tversky 1 showed that base rate information is underutilized. If "underutilized" implies "underestimated," then one might expect the conjunctive estimate would be less likely to fall beneath the low estimate of the first conjunct, and single conjunction errors would be plentiful. However, we assert that Kahneman and Tversky did not mean that the magnitudes of base rates were underestimated. Furthermore Morgan et al. 2 presented evidence that pre-test probabilities are usually overestimated, not underestimated. The research by Kahneman and Tversky clearly shows that "underutilized" means "insufficiently considered." If persons in Study 2 insufficiently considered the base rate (Component 1), then strong propensity of physicians to assign a conjunction estimate higher than a component exhibited in Studies 1 and 3 would be more likely to occur to Component 2, which would attract relatively more attention than it did in the other 2 studies in which the first component was not an insufficiently regarded base rate. This would result in the conjunction being more likely to exceed only Component 2 in Study 2 than in the other 2 studies. This is the result we obtained.